What would Dr. W. Edwards Deming do?

In a keynote called Beyond Checkboxes: Embedding True Cyber Resilience John Willis, a luminary in the DevOps world, shared his insights on the intersection of DevOps and the rapidly advancing field of artificial intelligence (AI). Willis began his keynote by invoking the wisdom of Dr. W. Edwards Deming, a figure who has deeply influenced his thinking for many years. Dr. Deming, renowned for his principles of quality and management, challenges us to think differently—a perspective crucial as we face the evolving landscape of DevOps and AI. Dr. Deming’s relevance to DevOps lies in his emphasis on continuous improvement, systemic thinking, and the scientific approach to management. These principles are foundational to addressing the complexities introduced by AI, as they encourage a holistic and adaptive mindset necessary for integrating advanced technologies into existing workflows.

DevOps Challenges in the AI Era

In our current digital era, software not only supports but defines businesses. AI, especially through generative models like LLaMA and GPT-4, has unlocked unprecedented capabilities in automation, data processing, and decision-making. However, as Willis emphasized with great power comes great responsibility. The integration of AI into organizations inevitably leads to significant challenges.

Technical Debt

One of the primary challenges AI introduces is the acceleration of technical debt. The rapid adoption of AI often leads to outdated models, poorly integrated solutions, and legacy systems struggling to keep up with new innovations. This technical debt can hinder progress and lead to inefficiencies.

Cybersecurity Vulnerabilities

AI’s role in cybersecurity is a double-edged sword. While AI can enhance threat detection and response, it also creates new vulnerabilities. AI models can be poisoned and deepfakes pose significant threats to organizational security. This dual nature requires careful management and robust security measures.

Operational Complexity

Managing AI-driven systems adds layers of operational complexity. Organizations must orchestrate diverse AI tools, ensure data integrity, and mitigate risks associated with AI model drift and biases. This complexity necessitates a reevaluation of existing DevOps processes to accommodate AI’s unique demands.

AI Hallucinations

A notable issue Willis highlighted is AI “hallucinations,” where models generate incorrect or misleading information. He advocated for the use of tools like Retrieval-Augmented Generation (RAG) to address this problem. RAG leverages vector databases to combine verified data with large language models, ensuring higher accuracy in AI outputs.

Enter the LORMA Stack: A Set of Best Practices to Navigate Challenges in the AI Revolution

In order to deal with these challenges Willis recommended adopting best practices which he coined the “LORMA stack”. The LORMA stack is derived from well known DevOps practices as well applying Deming’s ideas to the Pandora’s box that AI represents.

L – Language Model Orchestration

Orchestrate small and large language models in order enhance task performance while reducing computational costs. Look for strategies that leverage both SLMs and LLMs to mitigate the computational demands of LLMs.

O – Observability  

Enhance monitoring capabilities to include AI-specific metrics such as model accuracy, bias detection, and hallucinations. Think observability beyond tools like Honeycomb and Dynatrace.

R – Retrieval Augmented Generation

Incorporate Retrieval-Augmented Generation (RAG) techniques to address AI hallucinations. Use vector databases to integrate your data corpus with large language models, significantly improving the accuracy, and relevance of AI-generated responses

M – Model Providers

Understand how each of the model providers from OpenAI to MistralAI manage their embeddings, so the representation of real world objects like words and images in machine processable form. This allows you to make informed choices when it comes to the specific problems you want to solve.     

A – Autonomous Agent Management 

Think of one single AI as a task worker processing a specific low level task. The nascent field of autonomous agents concerns itself with higher level AIs which manage these tasks workers. Breakthroughs in this technology have the potential to truly revolutionize automation. 

Hands-on Intensivtrainings

DevOpsCon Camps zu Kubernetes, Service Mesh mit Istio und Monitoring

Mehr erfahren

The Main Takeaway: More DevOps, not less

John Willis’ opening keynote was uniquely insightful due to his integration of Dr. W. Edwards Deming’s principles with the current challenges and opportunities presented by AI in DevOps. He provided a holistic view, emphasizing the importance of continuous improvement, systemic thinking, and ethical AI practices. Willis left no doubt that the inevitable advance of AI in all our processes will create mountains of technical debt. And he was unequivocal that it is our duty as organizations to manage this debt. His forward-looking perspective on sustainable AI innovation is that more DevOps is required, not less.

Curious to know more? Dive into the full keynote and experience the insights firsthand